Re: Site Scaning & IP graps

• To: www-security@ns2.rutgers.edu
• Subject: Re: Site Scaning & IP graps
• From: Dennis Boone <drb@gopher.cl.msu.edu>
• Date: Fri, 22 Mar 96 15:45:09 -0500
• In-Reply-To: (Your message of Thu, 21 Mar 96 19:33:49 EST.) <Pine.SOL.3.91.960321193127.9027Y-100000@thebrain.aa.ans.net>

 >   Good spiders will ask for /robots.txt and find out what to do with 
 > themselves if they find it.
 > 
 >   Generally grepping for /robots.txt will give you a list of spiders that 
 > have found you.

The access log typically doesn't have the agent name, just the host name
that called for the file.  Some are obvious (fourteen.srv.lycos.com seems
to be visiting me today), others not (204.162.98.47 ??).

While it is possible to have all of the information in one log file with
the NCSA server, I'd bet most folks aren't using the feature.  I don't
know about other servers.

Dennis Boone
MSU CWIS Team

• Re: Site Scaning & IP graps
• From: "Brian W. Spolarich" <briansp@ans.net>

• Previous: SWISH/WWWWAIS access control gotchas?
• Next: Re: Site Scaning & IP graps
• Index(es):
  • Index
  • Thread